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ABSTRACT 


To evaluate the Navy's training under instructional systems 
development (ISD) procedures the annual, force-level, NATOPS 
examination scores of pilots in operational squadrons who had 
been trained under the recently installed ISD program at VP-3l 
were compared with those who had not been trained under ISD. 
Scores from only the open- and closed-book written examina- 
tions, which pertain to normal and emergency procedures and 
aircraft systems knowledge, were used since their content re- 
Flected most closely the course objectives of the nontactical 
EerOot tragmning provided by VP-31. Pilot rank, designation, 
time-in-squadron, and squadron location were used as controls. 
The type of training made no difference in open-book scores. 
For closed-book scores, ISD training resulted in a marginal 
but consistently negative difference as a main effect and in 
a significant negative difference when interactive with 
squadron location. At one location ISD-trained pilots scored 
Significantly lower while there was no difference among 
pilots at another location. The findings are discussed with- 
in the context of the study and with respect to training and 


ISD in general. 
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te LNERODUCT ION 


A. BACKGROUND 


In support of its aviation mission, the U.S. Navy operates 
training facilities located throughout the country to prepare 
both officers and enlisted personnel for jobs within the 
aviation community. One such facility is Patrol Squadron 
Thirty One (VP-31), which is the Navy's west coast P-3 Orion 
Replacement Air Group (RAG). VP-3l is responsible for, 
inter alia, the training of maintenance and flight crew 
personnel destined for operational P-3 squadrons located in 
California and Hawail. 

Prior to the introduction of modern instructional systems 
design (ISD) methods, VP-31 conducted pilot training utiliz- 
ing traditional methods and media consisting primarily of 
classroom lectures, examinations of comprehension, etc. 

Course objectives tended to be more content oriented --"famil- 
larize the student with performance charts contained in 
chapters ll and 12"-- rather than behaviorally oriented. 
Inherent in these traditional methods was the problem of 


measuring the effectiveness of training. 
B. INSTRUCTIONAL SYSTEMS DEVELOPMENT (ISD) 


Following World War II and the establishment of systems 


analysis as a scientific discipline, psychologists and 





educators recognized the possible benefits of applying 

systems technology to the field of instruction. These 
pioneers of modern instructional technology, who came not 

only from academia but also from quasi-governmental agencies 
such as RAND and HUMRRO, saw great potential in the use of 
techniques such as modelling to increase the precision, 
objectivity, and creativity of the judgment they exercised 

in solving the problems of course design [Montemerlo, 1979]. 
Perhaps the most significant contribution to the field of 
instruction by training technologists has been a demanding 
emphasis upon the precise definition of learning objectives 
and goal-oriented performance evaluation. Enabling objec- 
tives, final objectives, and course objectives form a hierarchy 
which serves to, among other things, provide a clear pathway 
for both students and instructors, and to answer the question-- 
“precisely what will a student be able to do after success- 
fully completing a course of instruction?" Goal-oriented 
performance evaluation provides a method to measure accurately 
the effectiveness of training programs that are oriented 
toward producing graduates capable of actually doing the 

TOD. 

Instruction can be divided into five categories represen- 
ting learned capabilities, i.e., verbal information, cog- 
nitive strategies, intellectual skills, motor skills, and 
attitudes [Gagne & Briggs, 1979]. Associated with each 


variety of learned capability are conditions for learning. 





Traditional instructional methods have always emphasized 
external conditions for learning such as repetition and stim- 
ulous-response reinforcement. Modern theories of learning 
focus also upon the internal conditions for learning such as 
prior learning, recall, and various levels of memory [Gagne © 
& Briggs, 1979]. 

Taxonomies such as the five types of learning described 
by Gagne serve to group similar objectives in a course design 
and provide for mutually applicable learning conditions. 
Additionally, grouping facilitates the employment of like 
methods of evaluation. Performance measures of a motor skill 
such as the manipulation of aircraft controls requires a 
student to perform bodily movements involving muscular activity. 
The performance measure appropriate to the learning of verbal 
information requires a student to state the relationship 
between two or more named objects or events either orally or 
Pewee ting. 

Attitudes are complex states of human beings relating to 
things, people, or events which can be described in terms of 
either their cognitive or affective aspects. Measurement of 
attitudes is most logically suited to the affective aspect 
and therefore centers upon observing a person's chosen al- 
ternative course of action from a selection of options. 

Intellectual skills form a hierarchy which starts with 
Simple discriminations and concrete concepts. Simple rules, 


which include defined concepts, follow discriminations and 
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lead to problem solving through higher-order rules. Perfor- 
mance measures of intellectual skills range from the basic 
act of distinguishing objects to the formulation and employ- 
ment of complex rules in order to achieve solutions to 
unique problems which are new to the student. 

Cognitive strategies are special forms of intellectual 
skills. These special skills actually represent processes 
by which learners modify their ways of attending, learning, 
remembering, and thinking. The salient difference between 
intellectual skills in general and cognitive strategies is 
the object of the skill. Cognitive strategies focus upon 
the learner's own thought process whereas intellectual skills 
relate to objects and events [Gagne & Briggs, 1979]. The 
performance measure of a cognitive strategy requires the 
novel solution to a problem where familiar solutions are 
not suitable. 

Goal-oriented performance objectives are requisite 
measures of learning success in ISD programs. However, the 
question of-- "performance quality" must be considered. 
Objective-referenced examinations employing a criterion- 
referenced interpretation provide the most acceptable pro- 
cedure for a variety of reasons. The validity of an objective- 
referenced test is directly related to its congruence with 
the course objective. Reliability of such a test is derived 
from the degree of consistency of repeated examinations. 


In an objective-referenced test of intellectual skills, 
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motor skills, or verbal information, mastery is a relevant 
concept and error-free performance can be achieved. However, 
cognitive strategies and attitudes are not suited to the 
mastery criterion because of their lack of an absolute 
measure [Gagne & Briggs, 1979]. 

Group instruction can be regarded in three separate 
categories which differ in the amount of precision afforded 
to the management of instructional events by the instructor. 
In a two-person tutorial situation, the instructor is pro- 
vided with the most control over instructional events. A 
small group is defined as three to eight people and is 
dependent upon group interaction to provide instructional 
events such as feedback, recall, etc. [Gagne & Briggs, 1979]. 
Large groups of more than 15 people are characterized by 
weak control over instructional events by the instructor. 
Accordingly, ina large group, learning is highly dependent 
upon the learner's own strategies of self instruction. To 
counter the inherent weak control of learning in large 
groups, the mastery concept for enabling objectives serves 
to diagnose learning performance early on and facilitate 
attainment of the course objectives in ISD programs. 

Individualized instruction is designed in the same 
fashion that individual lessons are designed for group in- 
struction. Objective hierarchies, learning events, and 
conditions for learning all apply to the design of modules 


SUlted to individualized instruction. The distinction lies 


ke 


: i 





in the delivery system employed for individualized instruction. 
Modules are normally self-paced and the materials do most of 
the teaching. More frequent feedback and progress checks are 
found in programs of individualized instruction. Under an 
ISD system, students are typically provided a series of 
modules which each contain: 

1. A performance objective 

2. A set of materials and learning activities 

3. A method of self-evaluation to determine mastery 


4. A provision for verification of learning to the 
instructor [Gagne & Briggs, 1979] 


As an organization responsible for a myriad of training 
programs ranging in subject matter from remedial reading to 
the conduct of war on a global basis, it was not at all 
surprising that the U.S. military establishment became in- 
terested and involved in the methods of ISD as it gained in 
credibility. With its basis in scientific methods and logical 
appeal, ISD became an attractive alternative to traditional 
training methods which offered the unique advantage of a 
capability to measure the effectiveness of a training program 
in terms of job-related performance. 

Today, ISD is an integral part of the training conducted 
by all four services. ISD has been employed by the U.S. 
military to ensure that: 

1. Courses are designed to teach only those tasks 

which, based upon objective field research and 
analysis of the tasks needed to be performed, 


the graduate will use and which can be taught 
in a formal training course. 
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2. Tests, the requisite for graduation, are accurate 
indicators of the ability to perform the required 
task [Department of Defense, 1980 p. XI-7]. 

The five phase ISD program, --i.e., analyze, design, 
develop, implement, and control-- and associated phase steps 
1s presented in Figure l. 

The Navy is committed to the systematic design of its 
training courses [Chief of Naval Education and Training (CNET), 
M75; Scaniand, 1974], The Interservice Procedures for In- 
structional Systems Development (Chief of Naval Education 
and Training) have been summarized by the Navy for its own 
use in a procedures manual entitled Procedures for Instruction- 
al Systems Development [CNET, 1978]. AF Pamphlet 50-58 [De- 
partment of the Air Force, July 1978] describes the Air Force 
ISD procedures and AF Manual 50-2 [Department of the Air 
Force, July 1975] promulgates the following policy regarding 
Paemutiikazarbion of ISD. 

1. Apply ISD to all new instructional systems 


2. Selectively apply ISD to existing instructional 
systems where economically feasible 


The extensive use of ISD by the military has prompted 
periodic review of training programs [Brock, 1977; Logicon 
Incorporated, 1978] . These studies emphasize the importance 
of job-related learning objectives as a prerequisite to 
successful ISD programs. If lists of learning objectives 
are provided for students, it seems natural, however, that 
their efforts in learning will be generally limited to 
those items on the list. Consequently, ISD program design- 
ers must leave no stone unturned in an exhaustive search 
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[CNAVEDTRA 110] 
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to identify all relevant items in a task analysis. Under 

ISD, it is naive to assume that the learning of important but 
easily overlooked items, such as a desired attitude or a 
knowledge of the relationship of components to a total system, 
will take place in the absence of specifically related 
learning objectives. 

Montemerlo [1979] cautions designers of instructional 
systems against a potential problem associated with the 
atomization approach of ISD manuals. He alleges that students 
inundated with a torrent of modularized learning objectives 
may MisS an overall goal. Atomistic descriptions tend to 
feed the learner a large number of new terms, definitions, 
and relationships faster than he can assimilate them [Monte- 
merlo, 1979]. The spiral approach to learning, which is 
the basis for most apprenticeship programs, focuses upon 
progressively more complex hands-on practice interspersed 
with limited book-learning. It is because of channel capacity, 
which is the amount of information which a person can process 
before becoming confused, that Montemerlo [1979] asserts 
Spiral learning works better than atomistic descriptions. 

Training is an investment in human capital and represents 
an organizational asset. Associated with a training invest- 
ment in human capital is an allocation of resources, such 
as time and money. The ISD method of instruction is extremely 
costly in terms of time required to develop and implement a 


program. Lexicon Incorporated, [1978] have tentatively 
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suggested an excess of five years is required, under ideal 
conditions, for the conversion of typical flying programs. 
However, ISD programs, with self-instruction, offer the 
potential of reducing the manhours spent by instructors in 
teaching. In order to determine accurately the total cost 
of an ISD application, the extensive costs related to 
development of the program must not be overlooked as these 
front-end costs are significant. 

The evaluation of a learning program is typically cen- 
tered upon three questions: 


1. To what extent have the stated objectives of 
instruction been met? 


2. In what ways and to what degree is it better than 
Eioeudbe (OL Inseruction) 2 will supplant? 


3. What additional, possibly unanticipated, effects 
has it had and to what extent are these better or 
worse than the supplanted unit? [Gagne & Briggs, 
1979] 

Formative evaluation is normally conducted during the 
development of a new system and serves to enhance the final 
product through modification or revision. By formative 
evaluation, quality in the design of a program can be ensured. 
Summative evaluation, on the other hand, is performed upon 
an existing course of instruction and normally serves to 
compare a new system to an "old one." During a summative 
evaluation, variables such as student aptitude for learning, 
the process of instruction, and learning support items must 


be controlled if a truly meaningful and unbiased evaluation 


7s tO Gesmit. 


18 





The training of Naval Aviators for duty as pilots and 
copilots in P-3 aircraft involves primarily the learning 
taxonomies of motor skills,verbal information, and intel- 
lectual skills. However, positive attitudes towards things 
such as safety and furthering aviator skills should also be 
considered germane to a total training program. There is 
an emerging acceptance of ISD as a total systems approach 
rather than a series of steps to be accomplished on some 
segment or portion of flying training. This acceptance is 
primarily responsible for the fact that ISD teams are 
beginning to return to the task analysis in order to 
strengthen it by means of a systematic, rather than a 
cursory or short-cut approach [Lexicon Incorporated, 1978, 
me 5S). 

The purpose of this study was to examine the effective- 
ness of the application of ISD to VP-31 pilot training as 
measured by scores achieved upon annually administered, 
force-level, NATOPS examinations. Another study currently 
underway [Fadness, 1980], has been designed to examine the 
application of ISD to the training of P-3 acoustic sensor 


Operators at VP-3l. 


1 





if. METHOD 


A. APPROACH 


In order to evaluate the effectiveness of pilot training 
at VP-~3l under a new ISD program, a sample of both ISD and 
non-ISD trained graduates was required. Under ideal circum- 
stances, an experimental group (ISD) and an identical control 
group (non-ISD) would be trained in parallel under conditions 
as identical as possible. Following graduation, the training 
received by both groups would be evaluated uSing identical, 
job-related performance measures. Further, retention of 
training could be measured at a later time. 

Under the constraints that this study experienced, the 
experimental and control groups had to be reconstructed from 
individuals already in operational squadrons located at two 
west-coast P-3 bases who had received training under one 
condition or the other. Since the training received by these 
individuals had occurred over a considerable period of time, 
the individuals varied extensively in both pre- and post- 
training experiences. Accordingly, demographic data were 
collected to control for these differences statistically. 
Scores achieved upon annually administered force level 
standardization examinations were selected as the dependent 
variable because of the relatively constant examination 


content over the time period examined. 
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B. DESCRIPTION OF ANNUALLY ADMINISTERED STANDARDIZATION 
EXAMINATIONS 


Standardization of normal and emergency operating proce- 
dures for crewmembers of Naval aircraft is provided by a 
program known as Naval Air Training and Operating Procedures 
Standardization (NATOPS). Included within the NATOPS program 
for P-3 Orion pilots is a squadron-level, initial certifica- 
tion to serve in the capacity of pilot-in-command and subsequent 
annual recertifications. Each certification process consists 
of two written examinations (one open book and one closed 
book) and an inflight check of piloting skills. Each phase 
of this examination process is designed to evaluate stated 
skill areas. Open-book examinations contain 24 multiple- 
choice questions relating primarily to performance charts 
located within the NATOPS manual. Pilots are required to 
compute various performance values from charts and select 
the correct answer to a written question from four choices. 
Closed-book examinations contain 40 multiple choice questions 
relating to normal and emergency procedures and aircraft 
systems. Pilots are required to respond to a written question 
by selecting one of four possible answers. Flight checks, 
which last approximately three hours, are designed to evaluate 
both knowledge of procedures and systems (verbal answers to 
verbal questions), and piloting skills. The content of written 
examinations must follow a distribution of subject matter 


-~-number of questions per subject-- specified by the force- 
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level evaluation team. To assist in examination construction, 
the force-level NATOPS evaluation team maintains a sizeable 
bank of sample questions for each subject area. Objective 
performance criteria for flight checks is provided within the 
NATOPS program to aid in standardization. 

In addition to squadron level NATOPS examinations, force 
level NATOPS examinations are administered to each operational 
Squadron On an annual basis. Force-level examinations consist 
of two written examinations (one open book and one closed 
book) for all crewmembers and flight checks for only three 
selected personnel per crew position. Force-level evaluations 
are intended to be similar in content to squadron-level 
evaluations. Crewmembers certified by the NATOPS force-level 
evaluator are awarded squadron NATOPS instructor status and 
are authorized to conduct squadron-level NATOPS check flights 
for their crew position. The results of force-level NATOPS 
unit evaluations for west-coast (California and Hawaii) P-3 
squadrons are maintained by the COMNAVAIRPAC P-3 NATOPS 
evaluation team. Force level NATOPS written examination 
scores achieved by P-3 pilots between 1978 and 1980 were 
made available for this study hy the Commanding Officer, 
Patrol Squadron Thirty One, and the COMNAVAIRPAC NATOPS 
pilot evaluator. It was agreed that the names of individual 
pilots and their respective squadrons would remain anonymous 


throughout the analysis. 
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Co orir LE DESCRIPTION 


Between January 1978 and May 1980, 31 west coast, opera- 
tional P-3 squadrons received force level, NATOPS, unit 
evaluations. In the Spring of 1979, graduates of VP-3l's new 
ISD training program began reporting to west-coast P-3 squadrons 
and participating in scheduled, force-level, NATOPS examina- 
tions. Three hundred eighty-two pilots from 10 squadrons 
located in California and Hawaii were selected as the initial 
sample. Composition of the initial sample was based upon two 
primary factors: 

l. Obtaining a maximum number of ISD-trained pilots 

2. The deployment status of squadrons which determined 

their ability to provide demographic data on sample 
Daocs 
Demographic data regarding the sample pilots were provided by 
squadrons through crew lists applicable to the date of the 
unit evaluation. 

During initial analysis it was not surprising to discover 
that because of the January 1979 implementation date of ISD 
at VP-3l, sample pilots trained under the ISD condition had 
generally been onboard their assigned operational squadrons 
Only a few months. Accordingly, the sample was reduced to 
those pilots with 12 or fewer months experience onboard their 
assigned squadrons. This reduction provided for a total of 
123 total pilots of which 61 were trained under the ISD 


program (Tables 1 and 2). 
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Table l 


Sample of Pilots Used in the Study by Rank, Months 


im the Seuadron, Type ef Training and Location* 


RANK MONTHS 
1-4 265 


Location A 


0-5 Za eels) 0 ( 1) 
0-4 ZED) 4 ( 2) 
0-3 er) Za (els) 
0-2 oget 12) SG) 
0-1 IL ay) 230) 
Sub-total ees ( 21) 16 (10) 

heocat1on.s 
0-5 Oko, OD) 
0-4 0 ( 3) 3) 
0-3 O( 1) i 
0-2 (7) 5 (a) 
0-1 Ome 1) Bera. 203) 
Sub-total deat 12) 8 (10) 
otal eS es 373)) PN ( 219\) 


* Numbers in parenthesis represent those who were trained 


Mace: the ISD condition 
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HP) 
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0) 
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0) 
0) 
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TOTAL 


3 
8 
5 
28 


47] 


gm 
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62 


( 3) 
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( 4) 
(20) 
2) 
(38) 


(aL) 
( 6) 
(oa) 
(14) 
on) 
(23) 
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RANK 


Sub-total 


Sub-total 


Total 


Table 2 


Sample of Pilots Used in the Study by Rank, 
Designation, Type of Training, and Location* 


NOP 


i 
5 
3 
8 
2 


de 7 


Sr uso - © 


Ov 


Ze 


( 0) 
no.) 
) 
(14) 
oe) 


(19) 
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(17) 
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PP3P PP2P 
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(0) 
(0) 


(0) 


(Ss 


Pee 


oO Oo O *. N 


Sora 2 oO F 


(2) 
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(0) 


(2) 
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* 
Numbers in parenthesis represent those who were 
the ISD Condition 
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D. DEPENDENT VARIABLES 


Force-level NATOPS written examination scores achieved 
by the 123 pilots during unit evaluations were selected as 
the dependent variable on which to evaluate phase A and B 
Pilot training conducted at vEoo ee The suitability of force- 
level written examinations as a training effectiveness 
measure is established by the close agreement between the 
course objectives of phase A and B pilot training and the 
Subject matter specified for testing in the P-3 NATOPS 
program. During phase A and B, pilot training subject matter 
consists of normal and emergency procedures and the operation 
of aircraft systems. It is during phase C and D, which have 
been purposely excluded from this study, that training in the 


various aspects of ASW and tactics is conducted. 


EE. INDEPENDENT VARIABLES 


Various demographic variables in addition to the type of 
training received were considered, and the following five were 
ultimately selected (See Table 3). 

1. Instructional Systems Development (ISD) 

The Instructional Systems Development variable iden- 
tifies the type of training that a pilot within the sample 
received while attending VP-31. Those pilots who were trained 
under the new system implemented in January of 1979 were 
coded 1, and those trained under the old system were coded 


Or 
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NAME 


Pabilreccs 


Definition of Variables 


VALUE 


LEGEND 


Pilot Rank 
(RANK) 


Designation 
(DESIG) 


Months 


(DMONTHS) 


Meaning 
(ISD) 
Squadron 
(DSQD) 


re F&F W NY FF MN F&F W DN FF 


Va aS & 


Ensign 

Lieutenant Junior grade 
Lieutenant 

Lieutenant Commander 

Commander 

Patrol Place Commander (PPC) 
Patrol Plane Second Pilot (PP2P) 
Patrol Plane Third Pilot (PP3P) 
Non-Designated Pilot (NOP) 


1-4 Months Onboard Squadron at Time 
of Test 


5-8 Months Onboard Squadron at Time 
of Test 


9-12 Months Onboard Squadron at Time 
of Test 


Non-ISD-Trained Pilot 

ISD-Trained Pilot 

All Squadrons Located at Location A 
All Squadrons Located at Location B 
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Zee eo Rank 
The rank of all sample pilots at the time they took 
the annual NATOPS examination was coded from 1 to 5 represent- 
ing the prescribed military structure of 0-1 through 0-5. 
3. Designation 
Pilots in the P-3 community are grouped into four 
general categories which are determined by their qualification 
as pilots. Nonrated pilots (NOP) are those pilots who have 
not yet qualified. Patrol Plane Third Pilots (PP3P) have 
successfully completed the first stage of the P-3 pilot 
Personal Qualifications Standards (PQS) training but have 
not yet completed a NATOPS check. Patrol Plane Second Pilots 
(PP2P) have successfully completed the second stage of PQS 
training and a pilot NATOPS check. A PP2P is considered 
"safe for solo" and may serve as pilot-in-command during a 
nontactical mission. Patrol Plane Commanders (PPC) have 
successfully completed the required PQS and the training 
directed by the Commander, Patrol Wings Pacific (CPWP). 
Patrol Plane Commanders are authorized to serve as pilot-in- 
command of tactical missions. 
4. Months 
The months variable represents the number of months 
that a pilot in the sample was onboard his squadron at the 
time of the force-level NATOPS examination. Although this 
variable ranged from 1 to 36, the sample, as stated, was 


restricted to those pilots with from 1 to 12 months onboard 
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a squadron when the examination was administered. This 
restriction reduced the sample size to 123 but provided for 
a more representative distribution of the ISD variable due 
to the newness of ISD at VP-3l. To facilitate analysis, 
aggregation of the months variable resulted in three groups: 
1-4 months, 5-8 months, and 9-12 months. 
5. Squadron 

The squadron variable served to distinguish the vari- 
ous squadrons to which west coast P-3 pilots are assigned 
Subsequent to VP-3l training. Aggregation of the squadron 
variable resulted in two groups representing the two bases 
at which squadrons are located. 

In the case of all variables, aggregation was identi- 


Fied by a preceding letter D. 


MeANALYSIS OF THE DATA 


Analysis of the data was conducted at three levels to 
investigate all potential aspects of the problem. Initial 
analysis focused upon the distribution of open- and closed- 
book scores by demographic variables. Additionally, tests of 
independence between the distribution of the subjects on the 
ISD variable and each of the demographic variables were com- 
puted and examined using chi-square. Utilizing further 
aggregated variables (see Results section), a 2x 2x 2 x 3 
factorial analysis of variance of open- and closed-book 


examination scores was conducted. This investigation utilized 


ao 





a Statistical Package for the Social Sciences [Nie, Hull, 
Jenkins, Stienbrenner & Bent, 1978] computer program and was 
designed to reveal the significance of four demographic 
variables (ISD, DSQD, DRANK, and DMONTHS) in accounting for 
the heterogeneity of open- and closed-book examination scores. 
Due to missing DDESIG data points in the sample (Table 2), 
the DDESIG variable was excluded from the factorial analysis 
of variance to permit the computation of two-way interactions. 
Further, in view of the high degree of dependence between 

the DDESIG and DRANK variables (Chi-square test of indepen- 
dence significant at <.001), exclusion of the DDESIG variable 
would not detract from the analysis. Three-way or higher 
interactions were pooled with the error sum of squares. Ag- 
gregated variables were again used for multivariate analysis 
of the variance among open- and closed-book examination 
scores considered simultaneously. This analysis was executed 
utilizing the Statistical Analysis System [SAS Institute, 
1979] computer program and examined the effects of all five 
demographic variables in accounting for the variance of com- 
bined scores. 

In addition to the statistical analyses described, 
Operational factors associated with the conduct of training 
at VP-3l were examined. Fiscal year averages of salient 
measures were compared for FY77, FY78 and FY79. Factors 
such as staff size, instructor complement, student load, time 
spent in ground school during the familiarization phase, and 
total syllabus length were considered germane to the study. 
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Pe eee CaS 


A. SUMMARY RESULTS 


Based upon the initially defined independent variables, 
means and standard deviations for open- and closed-book ex- 
aminations were computed by category of training received at 


VP-31 (See Table 4). 


B. AGGREGATED VARIABLES 


Further analysis of the problem was enhanced by aggrega- 
tion of the rank and designation variables to permit the use 
of multivariate procedures. The five-category rank variable 
was aggregated into two groups: 0-1 through 0-3 and 0-4 
maeougn 0-5. This distribution provided for not only a 
grouping of junior and senior officers, but additionally, it 
provided an identification of first- and second-tour pilots. 
The four level designation variable was combined to achieve 
two levels of designation: NOP and PP3P, and PP2P and PPC. 
This division provided a means to identify pilots who had 
completed a NATOPS certification check. The squadron vari- 
able and months variable were already aggregated (See Table 
3). Means and standard deviations for the newly aggregated 
rank and designation variables are presented in Table 5. 

Tables € and 7 present a contingency analysis of the 
training variable with each of the aggregated demographic 


variables to check for independence of the training variable 
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Table 4 


Means and Standard Deviations of Annual 
NATOPS Open and Closed Book Examinations 
by Category of Training Received* 


DEMOGRAPHIC VALUE eto. 
OPEN CLOSED 
NON- NON- 
FS. Sp ALL Sp Se ALL 
Pilot Rank 1 35 3.44 3. BO SS 37760 3.47 
~ 48 34 730 759 lg ~41 
(RANK) Zz 369 Sie ies Seay: 3.42 SrrGc Brn 3 
Pom -25 .28 ~39 ey 35 
3 3.60 Sol: 9.5)5 Sse 3.54 Oat 5S 
~32 53 ~44 239 23.0 34 
4 Bao) 3.89 Sa 3.74 ree a. Seu 
5 ike ley cold ~24 75/0 ~ 26 
5 4.00 See 3.90 3.68 oS Bie 
.00 - 16 v5 7 E26 aeat 
Designation l 3.94 ooo 390 Se 3 Seo. Sega 
.09 5S syleZ eae 20 5 AL 
(DESIG) 2 3.94 3.94 3.94 SoS B70 Se 
oO EO ~08 eS 26 a2 
3 3.74 See oa > 3.46 37459 ico o> 
~ 28 27 oe 3.5 . 30 23.1 
4 Seo 37760 3.64 3.44 3.64 Srey 
ore ~ 34 oo ~42 eS 7 
Months 1 37306 Se S765 3.43 S556 era 
«3.2 oe 34 ~44 Sel ~41 
(DMONTHS) 2 3483 Sree Be Sigeas 3605 S61 
Poo aa oe ee 229 250 
8 Sao ao 34079 33165 32.09 S07 
3 ~24 s25 .29 eS 2G 





Table 4 (continued) 
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Means above and standard deviations below in each set of 
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Table 6 


Contingency Analysis of the Distribution 
of ISD and Non-ISD Trained Pilots 
by Demographic Variables 


DEMOGRAPHIC 
VARIABLES 


DRANK 


DSQD 


DDESIG 


DMONTHS 


VALUES 


ToD 


42 
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38 


Zo 


ile 


a0 


5S 


20 


NON- 


ISD 


39 
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13 


47 
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3 
24 


25 
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Table 7 


Contingency Analysis of the Distribution 
of ISD and Non-ISD Trained Pilots 
by Aggregated Months and Location 


NON- er SIGNIFICANCE 
LOCATION DMONTHS iS) ISD SQUARE OF CHI SQUARE 
A , 1 2 ieZ Saer - al 
2 10 16 
3 a ik 
B 1 eZ a dl eeone - 00 
2 10 8 
3 a 6 
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distribution. There was a significant degree of dependence 
in the distribution of the ISD variable by DMONTHS at both 
squadron locations examined. This result was not surprising 
Since pilots trained under the new ISD system were more 
likely to have fewer months onboard their squadron than non- 
ISD trained pilots at the time of the unit NATOPS evaluations. 
The adjusted chi-square computed for the DMONTHS-ISD matrix 
was 17.8 and significant at the .0001 level. 


C. FACTORIAL ANALYSIS OF VARIANCE OF OPEN BOOK SCORES 
(TABLE 8) 


The DRANK, DSQD, and DMONTHS variables were found to be 
Significant as main effects in accounting for the observed 
heterogeneity among sample open-book NATOPS examination 
scores. Additionally, the interaction between DSQD and 
DRANK variables was determined to be significant. Mean 
open-book examination scores for both values of the DRANK 
variable were demonstrated to be nearly identical at loca- 
tion B and quite different at location A. Accordingly, 
although the DRANK and DSQD variables were significant when 
considered separately, the predictive power of the DRANK 
variable was enhanced by knowing the location variable 
(See Table 9 and Figure 2). It is interesting to note that 
in spite of the significant degree of dependence in the 
distribution of ISD by DMONTHS, the two-way interaction 
between these variables in the factorial analysis was not 
Significant. Further, ISD, per se, was not identified as 


a significant effect in this analysis. 
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DEMOGRAPHIC VARIABLE 


ISD 
DMONTHS 
DRANK 


DSQD 


Two Way Interactions 


ISD 
ISD 
ISD 
DMONTHS 
DMONTHS 


DSQD 


* 
Total D/F 


DMONTHS 


DSQD 
DRANK 
DSQD 
DRANK 


DRANK 


= 122 
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Table 9 


Mean Open-Book NATOPS Scores for Location (DSQD) 
by Rank (DRANK) 


Value of DRANK Location 
A B 
it Beaoe: B32 
2 3.9 a3: 
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4.0 EO@cation A 
_ Location B 


SCORE Bead) 


Figure 2. Interactive Effects of Rank and Location 
on Open-Book NATOPS Examination Scores 
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D. FACTORIAL ANALYSIS OF THE VARIANCE OF CLOSED-BOOK 
SCORES (TABLE 10) 


The factorial analysis of variance of closed-book scores 
identified the DMONTHS and DRANK variables as significant 
main effects. Although not significant at the .05 level, 
the ISD variable was significant at the .08 level. Further, 
the two-way interactions of ISD with DSQD and DSQD with 
DRANK were significant in accounting for score variance. 
Details of these interactions are shown in Table 11 and 
Figure 3. Pilots stationed at location B were observed to 
achieve nearly idential mean scores on the NATOPS closed- 
book examinations regardless of the type of training received 
while at VP-31 or their current rank. However, the mean 
scores achieved by pilots stationed at location A varied 
considerably by the type of training received while at VP-31; 
the non-ISD-trained group had a higher mean score than the 
ISD-trained group. The difference in these two means, as 
measured with a t test, was significant at the .01 level. 
Additionally, although the DRANK variable made no difference 
in the mean closed-book score at location B, it did account 
for a meaningful difference in mean scores at location A in 
that the pilots with higher rank, and presumably more exper- 
ience, made better scores. 

E. MULTIVARIATE ANALYSIS OF VARIANCE OF OPEN=- AND CLOSED- 

BOOK NATOPS SCORES (TABLE 12) 


A multivariate analysis of open- and closed-book NATOPS 
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Table 10 


Factorial Analysis of Variance of Closed-Book 
NATOPS Scores by Demographic Variables 


DEMOGRAPHIC VARIABLE 
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DSQD 


DRANK 


Two Way Interactions 


fSD DMONTHS 
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DMONTHS DRANK 


DSQD DRANK 


* 
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Table ll 


Mean Closed-Book NATOPS Scores for Location 
(DSQD) by Training (ISD) and Rank (DRANK) 


VARIABLE VALUE LOCATION 
A 
ISD 0 Sor 
ah 3.48 
DRANK 1 2) 5 20, 
2 Bye 3 
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Score 


Score 


4.0 Location A 


3.0 
(a) 
20 1 2 
DRANK 
4.0 
Location B 
Location A 
20 
(b) 
2.0 0 i: 
ne) 


Interactive Effects of Location (DSQD) 
with (a) DRANK and (b) ISD on Closed- 
Book NATOPS Examination Scores 
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Table 12 


Multivariate Analysis of Variance of Annual NATOPS 
Open- and Closed-Book Examination Scores 


HYPOTHESIS OF NO OVERALL 


Meeicl BY: pe SIGNIFICANCE OF F 
co} 20> es 
DRANK 4.6/7 2 Oue 
DMONTHS 253 -.04 
DDESIGN ORGS 294 
DSQD SO OZ 


*F Determined by Wilk's criterion where: 


F= Ee x ANEHQIP OL) with 2P and 2 (NE+Q-P-1) d/f 
L 

r = DET(E) 
Pewee (H+) 

H = sum of squares and cross product matrix for the null 
hypothesis 

E = error sum of squares and cross product matrix 

P = number of dependent variables 


Q = degree of freedom of null hypothesis 


NE = D/F of error 
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examination scores, considered simultaneously, identified 

the DRANK, DMONTHS, and DSQD variables as significant in 
accounting for score heterogeneity. Because of the previous- 
ly discussed sensitivity of open- and closed-book NATOPS 
scores to the experience variables (DRANK and DMONTHS) and 
the squadron location variable (DSQD), the multivariate 
analysis was performed to test the power of these effects 

on the combined scores. Although the correlation between 
open- and closed-book scores was computed to be only .33, 

the effects of squadron location and both experience vari- 
ables were again statistically significant in the multi- 
variate analysis. All interactive effects were excluded 

from this test because of a singular matrix caused by missing 
data points for the DDESIG variable which prohibited the 
computation of two-way and higher interactions. As in the 
factorial analyses, the ISD variable was not statistically 


Significant. 


F. OPERATING DATA COMPARISON (TABLE 13) 


The most noticeable changes in operational factors over 
the time frame examined (FY77 through FY79) have occurred 
in the following four areas: 

i Moeueent/instructor ratio 

Pease cona cour pilot load 

3. Ground school time 


4. Enlisted staff size 
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Table 13 


A Comparison of Operating Data for VP-31l Training 


FACTOR TYRPESOF FAGTOR Bava 7 FY78 FY79 
Student Instructor vere Aer eG io 5 
Ratio 
Aircraft Flights Perse LOuUL 8 5 
per Student Seconau Tour 6 5 
Simulator Periods eget. Poul ) 7 
per Student Second Tour 0 6 
Allotted Ground beste hour 196 Zo 
School Time (Hrs) Second Tour 162 ZO 
Syllabus Length Pest Tour 9 wks 10 wks 
3 days 
Second Tour 8 wks 6 wks 
2 days 6 days 
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As illustrated in Table 13, the student/instructor ratio was 
increased between the pre-ISD period of FY77 and FY78 and the 
ISD period of FY79 by approximately .39. This increase 
occurred concurrently with an expansion of the second-tour 
pilot load and would indicate that the efficiency of self- 
instruction associated with ISD was being realized. Ground 
school time specified for the familiarization phase was 
increased between FY77 and FY79 due to the introduction of 
the flight simulator and the inclusion of simulator time in 
ground school totals. In aggregate, operational data seem 
to have remained essentially constant over the time frame 
examined, except for a modest, but steady, increase in the 


enlisted staff size. 
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iV SCUSS ION 


The preceding analysis suggests that VP-31 pilot training 
under the new ISD condition serves to account only for lower 
scores on force-level, closed-book, NATOPS examinations by 
pilots assigned to one, of two, west-coast P-3 bases. Further, 
this study identified rank, squadron experience, and squadron 
location as statistically significant effects in accounting 
for the variance of examination scores. These results raise 
three salient questions Beceede ne P-3 pilot training and sub- 


sequent NATOPS evaluation. 


1. Is the pilot training provided by VP-3l1 in conformance 
with the best and proper application of ISD? 


2. Is the force-level NATOPS testing program an ap- 
propriate measure of the effectiveness of VP-3l 
Di lete training? 

3. Is ISD a more effective system for teaching pilots 
to learn, retain, and apply a knowledge of P-3 
Operating procedures and systems than the previous 
traditional system? 

To determine the degree of conformance between current 
training at VP-3l1 and the best and proper application of ISD 
would require a formative evaluation, the magnitude of which 
is beyond the scope of this study. On the surface however, 
it appears that with but a few exceptions, such as the lack 
of entry level testing and a true self-paced learning program 
which would allow students to complete training at a self 


determined schedule, the basic principles of ISD are being 


followed. Courseware Incorporated, a private-sector ISD 
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consultant, has been largely responsible for the design of 
VP-31"s new training program. 

The appropriateness of the force-level NATOPS testing 
program as a measure of VP-31 pilot training effectiveness 
was established by the close agreement between course objec- 
tives and the areas specified for evaluation in the NATOPS 
program. (See Appendix A for a detailed listing.) Although 
a certain degree of latitude is permitted in the construction 
of individual examination questions, the specific subject 
matter is clearly defined for both open- and closed-book 
examinations. The open-book examination is a particularly 
appropriate measure of training effectiveness because in 
general it requires pilots to compute values associated with 
aircraft performance which are identical to those required on 
the job. Analysis of the variance of open-book examination 
scores suggests that among the variables considered, the 
effects of experience (rank and months) and squadron location 
are statistically significant. Further, the effect of the 
teaining Variable (ISD or non-ISD) was not Significant. 

Closed-book examinations measure primarily verbal infor- 
mation and intellectual skills associated with operating 
procedures and aircraft systems. Since the examination re- 
quires pilots to first read and interpret a question before 
selecting a written answer, the relationship to on-the-job 
requirements is not as close as in the open-book examination. 


Analysis of the closed-book examination scores revealed, as 
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with open-book scores, a sensitivity to the rank and months 
variables. The type of training received was significant at 
the .08 level as a main effect and at the .03 level when 
interacted with the squadron location variable (ISD trained 
pilots scored lower than traditionally trained pilots at 
location A). The direction and consistency of the main effect 
is very obvious in Table 4. Of the 14 contrasts, in all but 
one the ISD=-trained pilots have lower closed-book examination 
scores. 

The preceding discussion suggests that organizational 
climate, as well as experience, is strongly related to pilot 
performance on NATOPS examinations. Formal syllabus training 
received at VP-31, either ISD or traditional, serves to 
provide pilots with a basic knowledge of P-3 systems and 
procedures. Retention of these skills and further learning 
takes place in the field at the squadron level where previous- 
ly learned skills have the opportunity to grow or deteriorate. 
Under a positive climate, learning is reinforced and perfor- 
mance 1s rewarded. Measurement of the learning climate 
associated with an organization should be based upon the 
attitudes of individuals which result from choices, such as 
whether or not to maintain previous learning and further 
develop aviator skills. 

The ISD method focuses sharply upon the development of a 
precise list of learning objectives based upon task analyses. 


Although the logic of the atomized approach toward learning 
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specified in design manuals is sound, course designers must 
exercise particular attention to detail when assembling task 
inventories and subsequent learning objectives. If graduates 
are deficient in an area not listed among the objectives, the 
problem is with the list and not with the program. The 
learning of attitudes, which is not emphasized within the 
Navy's ISD design manual, should be considered by course 
designers as a desired outcome of pilot training. 

This study seems to indicate that perhaps a truly positive 
attitude toward the further development of knowledge in the 
squadron environment is not being learned as well under the 
ISD system as it was under the traditional methods. Something 
may have been overlooked in the time-consuming atomization 
process of ISD when learning became limited to those items 
on the list. Additionally, emphasis on the individualization 
of instruction and self-instruction may have served to 
eliminate the positive effects of peer and instructor inter- 


actions in the learning of favorable attitudes. 
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V. CONCLUSION 


This study suggests that by utilizing force-level NATOPS 
examination scores as criteria for the effectiveness of pilot 
training conducted at VP-31, enhancement by the application 
of ISD has not yet been achieved. The following findings are 
germane: 

1. A reduction in closed-book NATOPS examination scores 
associated with ISD trained pilots, particularly at one of 
two west-coast P-3 bases, was identified. Although these ISD 
trained pilots did score lower on average than their tradition- 
ally trained counterparts, it 1S somewhat presumptuous to 
attribute the difference to the process of ISD until a forma- 
tive evaluation is conducted at VP-31 to establish the degree 
to which the pilot-training program conforms to the best and 
pioper utilization of ISD. 

2. Upon completion of a formative evaluation, further 
summative evaluation, under the controlled conditions of 
parallel training previously discussed, is required to 
determine the effectiveness of a program verified to be a 
proper application of ISD. An evaluation of this type is 
necessary to justify the extensive cost in terms of manpower 
and money allocated to the development of ISD at VP-3l. 

3. Pilot training provided by VP-31 during the time 
frame examined (either ISD or non-ISD) seems to have a 


limited effect upon force-level NATOPS examination scores. 
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However, the effects of squadron location and experience (rank 
and months) are statistically significant in accounting for 


the variance of NATOPS scores. 
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APPENDIX A 


A COMPARISON OF TRAINING OBJECTIVES AND THE SUBJECT 
MATTER OF NATOPS EXAMINATIONS 


Examination Subject 


Open 


Ordnance/Zero Fuel 
Weight Computation 


Fuel Density/Minimum 
Fuel for Flight 


Center of Gravity 


Limits 


Minimum Control Speed 


Runway Wind Components 


Refusal Speed 


ce, 6 RCR 

wea, Vlof, V50 (3) 
Distance to Liftoff 
Stall Speed/Stall 
Buffet 

Climb Performance 
Altitude Selection 


Max Range Airspeed 


Max Range SHP 


Training Objective 
Book Examination 


Define and Compute 
Zero Fuel Weight 


Utilize Minimum Fuel 
Flight Chart 


Define and Compute 
Center of Gravity, 
State Limits 


Compute VMC 


Compute Runway Wind 


Component 
Compute Refusal Speed 


Correction to 
Speed for RCR 


Compute 
Refusal 


Compute 
Compute distance to 
Tabet ois 

Compute Stall Speed 
Compute 3/4 Eng Climb- 
O@EM Light. Pach 


Compute Max Range 
Altitude 


Compute Max Range Air- 
speed 


Compute Max Range SHP 
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Take Off Speeds 
VO, VLOE, V50 Mead Eng) 


Unit  Beccon +4 


a 8 
al 8 
i 8 
SI i 
3 i. 
3 1 
5 i 
5 ar 
3 1 
3 1 
3 2 
3 & 
3 6 
3 6 





Examination subject 


Max Range Fuel 
Planning 


Loiter Time Predic- 
gon 


Angle of Attack 
Composite Cruise 
Flight Performance 
Chart 

Ditch Speed 


Available Range 
Remaining 


Landing Ground Roll 
Distance 


NATOPS Manual/ 
Evaluations 


Emergencies 


Training Objective 


Utilize Max Range 
Charts 


Compute Loiter Time 
Available 


Define AOA 
Utilize Composite 
Cruise Performance 


Chart 


Compute Ditch Speed 


Compute Available Range 


Remaining 


Compute Landing Ground 
Roll 


NATOPS Program Review 


Emergencies covered in 


Closed Book Examination 


Aircraft General 


Power Plants 


Propeller 


Electrical System 


Hydraulic System 


Auxiliary Power Unit 


Generaleaircrarte 


Describe the Operation 
and Limits of the 
Engines 


Describe the Operation 
and Limits of the 
Propeller 


Describe the Operation 
and Limits of the 
Electrical System 


Describe the Operation 
and Limits of the 
Hydraulic System 
Describe the Operation 
and Limits of the APU 
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nas 


Lesson # 


25 


12 


all 


elo, 
i) 


aes. 





Examination Subject Training Objective Unit Lesson # 


Flight Controls Describe the Operation 1 6 
Obrien wait Controls 
Pucomatic Flight Describe the Operation a 25,27 
Control System and Limits of the 
AUEeMadtcre Filzghte Control 
system 
Air Conditioning and Describe the Operation 1 18,19 
Pressurization and Limits of the Air 


Conditioning and 
Pressurization System 


Fuel System Describe the Operation 1 11,14 
and Limits of the Fuel 
System 

Radio and Nav Aids Describe the Operation i Bale ice 
of the Radios and Nav 23724, 
Aids Ze 

All Weather Systems Describe the Operation 1 20 


and Limits of the Foul 
Weather Systems 


Armament Covered in Tactics Section 
Instruments Describe the function iL 25 
and limits of cockpit 
Iinsecunents 
Weight and Balance Define and Compute Weight l 8 


and Balance Data 


Aircraft Performance Compute Max Range Cruise 3 6 
and Cruise Control 


Ditching and Bailout Describe Procedures for 4 12 
Ditch and Bailout 


Normal and Emergency Normal and Emergency 2&4 all 
Operating Procedures Operating Procedures 
Limitations Normal and Emergency 2&4 all 


Operating Procedures 


5/7 





‘ S ees 


“9 | vr? 


10. 


alee 


EZ). 
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